{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Face Mask Detection using NVIDIA TLT "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The MIT License (MIT)\n",
    "\n",
    "Copyright (c) 2019-2020, NVIDIA CORPORATION.\n",
    "\n",
    "Permission is hereby granted, free of charge, to any person obtaining a copy of\n",
    "this software and associated documentation files (the \"Software\"), to deal in\n",
    "the Software without restriction, including without limitation the rights to\n",
    "use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of\n",
    "the Software, and to permit persons to whom the Software is furnished to do so,\n",
    "subject to the following conditions:\n",
    "\n",
    "The above copyright notice and this permission notice shall be included in all\n",
    "copies or substantial portions of the Software.\n",
    "\n",
    "THE SOFTWARE IS PROVIDED \"AS IS\", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR\n",
    "IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS\n",
    "FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR\n",
    "COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER\n",
    "IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN\n",
    "CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## DetectNet_v2 with ResNet-18 example usecase\n",
    "\n",
    "The goal of this notebook is to utilize NVIDIA TLT to train and make Face Mask detection model deploy ready.\n",
    "While working on such application, this notebook will serve as an example usecase of Object Detection using DetectNet_v2 in the Transfer Learning Toolkit.\n",
    "\n",
    "0. [Set up env variables](#head-0)\n",
    "1. [Prepare dataset and pre-trained model](#head-1)\n",
    "    1. [Download dataset and convert in KITTI Format](#head-1-1)\n",
    "    1. [Prepare tfrecords from kitti format dataset](#head-1-2)\n",
    "    2. [Download pre-trained model](#head-1-3)\n",
    "2. [Provide training specification](#head-2)\n",
    "3. [Run TLT training](#head-3)\n",
    "4. [Evaluate trained models](#head-4)\n",
    "5. [Prune trained models](#head-5)\n",
    "6. [Retrain pruned models](#head-6)\n",
    "7. [Evaluate retrained model](#head-7)\n",
    "8. [Visualize inferences](#head-8)\n",
    "9. [Deploy](#head-9)\n",
    "    1. [Int8 Optimization](#head-9-1)\n",
    "    2. [Generate TensorRT engine](#head-9-2)\n",
    "10. [Verify Deployed Model](#head-10)\n",
    "    1. [Inference using TensorRT engine](#head-10-1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![Face Mask Detection Output](https://raw.githubusercontent.com/NVIDIA-AI-IOT/face-mask-detection/master/images/face-mask-detect-output.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 0. Set up env variables <a class=\"anchor\" id=\"head-0\"></a>\n",
    "When using the purpose-built pretrained models from NGC, please make sure to set the `$KEY` environment variable to the key as mentioned in the model overview. Failing to do so, can lead to errors when trying to load them as pretrained models.\n",
    "\n",
    "*Note: Please make sure to remove any stray artifacts/files from the `$USER_EXPERIMENT_DIR` or `$DATA_DOWNLOAD_DIR` paths as mentioned below, that may have been generated from previous experiments. Having checkpoint files etc may interfere with creating a training graph for a new experiment.*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Setting up env variables for cleaner command line commands.\n",
    "print(\"Update directory paths if needed\")\n",
    "%env KEY=tlt_encode\n",
    "# User directory - pre-trained/unpruned/pruned/final models will be saved here\n",
    "%env USER_EXPERIMENT_DIR=/home/detectnet_v2 \n",
    "# Download directory - tfrecords will be generated here\n",
    "%env DATA_DOWNLOAD_DIR=/home/data_fm_0916           \n",
    "# Spec Directory\n",
    "%env SPECS_DIR=/home/detectnet_v2/specs   \n",
    "# Number of GPUs used for training\n",
    "%env NUM_GPUS=1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1. Prepare dataset and pre-trained model <a class=\"anchor\" id=\"head-1\"></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### A. Download dataset and convert in KITTI Format <a class=\"anchor\" id=\"head-1-1\"></a>\n",
    "\n",
    "In this experiment we will be using 4 different datasets; \n",
    "\n",
    "1. Faces with Mask:\n",
    "    - Kaggle Medical Mask Dataset [Download Link](https://www.kaggle.com/ivandanilovich/medical-masks-dataset-images-tfrecords)\n",
    "    - MAFA - MAsked FAces [Download Link](https://drive.google.com/drive/folders/1nbtM1n0--iZ3VVbNGhocxbnBGhMau_OG)\n",
    "2. Faces without Mask:\n",
    "    - FDDB Dataset [Download Link](http://vis-www.cs.umass.edu/fddb/)\n",
    "    - WiderFace Dataset [Download Link](http://shuoyang1213.me/WIDERFACE/)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Download the data using provided links, such that all images and label files are in one folder. We expect in structure noted in GitHub repo.\n",
    "- Convert dataset to KITTI format \n",
    "- Use KITTI format directory as \"$DATA_DOWNLOAD_DIR\"\n",
    "\n",
    "\n",
    "Note: We do not use all the images from MAFA and WiderFace. Combining we will use about 6000 faces each with and without mask"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### B. Prepare tf records from kitti format dataset <a class=\"anchor\" id=\"head-1-2\"></a>\n",
    "\n",
    "* Update the tfrecords spec file to take in your kitti format dataset\n",
    "* Create the tfrecords using the tlt-dataset-convert \n",
    "\n",
    "*Note: TfRecords only need to be generated once.*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(\"TFrecords conversion spec file for kitti training\")\n",
    "!cat $SPECS_DIR/detectnet_v2_tfrecords_kitti_trainval.txt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Creating a new directory for the output tfrecords dump.\n",
    "print(\"Converting Tfrecords for kitti trainval dataset\")\n",
    "!tlt-dataset-convert -d $SPECS_DIR/detectnet_v2_tfrecords_kitti_trainval.txt \\\n",
    "                     -o $DATA_DOWNLOAD_DIR/tfrecords/kitti_trainval/kitti_trainval"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!ls -rlt $DATA_DOWNLOAD_DIR/tfrecords/kitti_trainval/"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### C. Download pre-trained model <a class=\"anchor\" id=\"head-1-3\"></a>\n",
    "Download the correct pretrained model from the NGC model registry for your experiment. Please note that for DetectNet_v2, the input is expected to be 0-1 normalized with input channels in RGB order. Therefore, for optimum results please download models with `*_detectnet_v2` in their name string. All other models expect input preprocessing with mean subtraction and input channels in BGR order. Thus, using them as pretrained weights may result in suboptimal performance. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# List models available in the model registry.\n",
    "!ngc registry model list nvidia/tlt_pretrained_detectnet_v2:*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create the target destination to download the model.\n",
    "!mkdir -p $USER_EXPERIMENT_DIR/pretrained_resnet18/"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Download the pretrained model from NGC\n",
    "!ngc registry model download-version nvidia/tlt_pretrained_detectnet_v2:resnet18 \\\n",
    "    --dest $USER_EXPERIMENT_DIR/pretrained_resnet18"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!ls -rlt $USER_EXPERIMENT_DIR/pretrained_resnet18/tlt_pretrained_detectnet_v2_vresnet18"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2. Provide training specification <a class=\"anchor\" id=\"head-2\"></a>\n",
    "* Tfrecords for the train datasets\n",
    "    * In order to use the newly generated tfrecords, update the dataset_config parameter in the spec file at `$SPECS_DIR/detectnet_v2_train_resnet18_kitti.txt` \n",
    "    * Update the fold number to use for evaluation. In case of random data split, please use fold `0` only\n",
    "    * For sequence-wise split, you may use any fold generated from the dataset convert tool\n",
    "* Pre-trained models\n",
    "* Augmentation parameters for on the fly data augmentation\n",
    "* Other training (hyper-)parameters such as batch size, number of epochs, learning rate etc."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!cat $SPECS_DIR/detectnet_v2_train_resnet18_kitti.txt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3. Run TLT training <a class=\"anchor\" id=\"head-3\"></a>\n",
    "* Provide the sample spec file and the output directory location for models\n",
    "\n",
    "*Note: The training may take hours to complete. Also, the remaining notebook, assumes that the training was done in single-GPU mode. When run in multi-GPU mode, please expect to update the pruning and inference steps with new pruning thresholds and updated parameters in the clusterfile.json accordingly for optimum performance.*\n",
    "\n",
    "*Detectnet_v2 now supports restart from checkpoint. Incase, the training job is killed prematurely, you may resume training from the closest checkpoint by simply re-running the same command line. Please do make sure to use the same number of GPUs when restarting the training.*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!tlt-train detectnet_v2 -e $SPECS_DIR/detectnet_v2_train_resnet18_kitti.txt \\\n",
    "                        -r $USER_EXPERIMENT_DIR/experiment_dir_unpruned \\\n",
    "                        -k $KEY \\\n",
    "                        -n resnet18_detector \\\n",
    "                        --gpus $NUM_GPUS"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print('Model for each epoch:')\n",
    "print('---------------------')\n",
    "!ls -lh $USER_EXPERIMENT_DIR/experiment_dir_unpruned/weights"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4. Evaluate the trained model <a class=\"anchor\" id=\"head-4\"></a>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!tlt-evaluate detectnet_v2 -e $SPECS_DIR/detectnet_v2_train_resnet18_kitti.txt\\\n",
    "                           -m $USER_EXPERIMENT_DIR/experiment_dir_unpruned/weights/resnet18_detector.tlt \\\n",
    "                           -k $KEY"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 5. Prune the trained model <a class=\"anchor\" id=\"head-5\"></a>\n",
    "* Specify pre-trained model\n",
    "* Equalization criterion (`Applicable for resnets and mobilenets`)\n",
    "* Threshold for pruning.\n",
    "* A key to save and load the model\n",
    "* Output directory to store the model\n",
    "\n",
    "*Usually, you just need to adjust `-pth` (threshold) for accuracy and model size trade off. Higher `pth` gives you smaller model (and thus higher inference speed) but worse accuracy. The threshold to use is depend on the dataset. A pth value `5.2e-6` is just a start point. If the retrain accuracy is good, you can increase this value to get smaller models. Otherwise, lower this value to get better accuracy.*\n",
    "\n",
    "*For some internal studies, we have noticed that a pth value of 0.01 is a good starting point for detectnet_v2 models.*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create an output directory if it doesn't exist.\n",
    "!mkdir -p $USER_EXPERIMENT_DIR/experiment_dir_pruned"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(\"Change Threshold (-pth) value according to you experiments\")\n",
    "\n",
    "!tlt-prune -m $USER_EXPERIMENT_DIR/experiment_dir_unpruned/weights/resnet18_detector.tlt \\\n",
    "           -o $USER_EXPERIMENT_DIR/experiment_dir_pruned/resnet18_nopool_bn_detectnet_v2_pruned.tlt \\\n",
    "           -eq union \\\n",
    "           -pth 0.8 \\\n",
    "           -k $KEY"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!ls -rlt $USER_EXPERIMENT_DIR/experiment_dir_pruned/"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 6. Retrain the pruned model <a class=\"anchor\" id=\"head-6\"></a>\n",
    "* Model needs to be re-trained to bring back accuracy after pruning\n",
    "* Specify re-training specification with pretrained weights as pruned model.\n",
    "\n",
    "*Note: For retraining, please set the `load_graph` option to `true` in the model_config to load the pruned model graph. Also, if after retraining, the model shows some decrease in mAP, it could be that the originally trained model, was pruned a little too much. Please try reducing the pruning threshold, thereby reducing the pruning ratio, and use the new model to retrain.*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Printing the retrain experiment file. \n",
    "# Note: We have updated the experiment file to include the \n",
    "# newly pruned model as a pretrained weights and, the\n",
    "# load_graph option is set to true \n",
    "!cat $SPECS_DIR/detectnet_v2_retrain_resnet18_kitti.txt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Retraining using the pruned model as pretrained weights \n",
    "!tlt-train detectnet_v2 -e $SPECS_DIR/detectnet_v2_retrain_resnet18_kitti.txt \\\n",
    "                        -r $USER_EXPERIMENT_DIR/experiment_dir_retrain \\\n",
    "                        -k $KEY \\\n",
    "                        -n resnet18_detector_pruned \\\n",
    "                        --gpus $NUM_GPUS"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Listing the newly retrained model.\n",
    "!ls -rlt $USER_EXPERIMENT_DIR/experiment_dir_retrain/weights"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 7. Evaluate the retrained model <a class=\"anchor\" id=\"head-7\"></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This section evaluates the pruned and retrained model, using `tlt-evaluate`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!tlt-evaluate detectnet_v2 -e $SPECS_DIR/detectnet_v2_retrain_resnet18_kitti.txt \\\n",
    "                           -m $USER_EXPERIMENT_DIR/experiment_dir_retrain/weights/resnet18_detector_pruned.tlt \\\n",
    "                           -k $KEY"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 8. Visualize inferences <a class=\"anchor\" id=\"head-8\"></a>\n",
    "In this section, we run the `tlt-infer` tool to generate inferences on the trained models. To render bboxes from more classes, please edit the spec file `detectnet_v2_inference_kitti_tlt.txt` to include all the classes you would like to visualize and edit the rest of the file accordingly."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For this you will need to create `test_images` directory containing at least 8 images with masked and no-masked faces, it can be from test data or simply face captures from your own photos. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Running inference for detection on n images\n",
    "!tlt-infer detectnet_v2 -e $SPECS_DIR/detectnet_v2_inference_kitti_tlt.txt \\\n",
    "                        -o $USER_EXPERIMENT_DIR/tlt_infer_testing \\\n",
    "                        -i $DATA_DOWNLOAD_DIR/test_images \\\n",
    "                        -k $KEY"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `tlt-infer` tool produces two outputs. \n",
    "1. Overlain images in `$USER_EXPERIMENT_DIR/tlt_infer_testing/images_annotated`\n",
    "2. Frame by frame bbox labels in kitti format located in `$USER_EXPERIMENT_DIR/tlt_infer_testing/labels`\n",
    "\n",
    "*Note: To run inferences for a single image, simply replace the path to the -i flag in `tlt-infer` command with the path to the image.*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Simple grid visualizer\n",
    "import matplotlib.pyplot as plt\n",
    "import os\n",
    "from math import ceil\n",
    "valid_image_ext = ['.jpg', '.png', '.jpeg', '.ppm']\n",
    "\n",
    "def visualize_images(image_dir, num_cols=4, num_images=10):\n",
    "    output_path = os.path.join(os.environ['USER_EXPERIMENT_DIR'], image_dir)\n",
    "    num_rows = int(ceil(float(num_images) / float(num_cols)))\n",
    "    f, axarr = plt.subplots(num_rows, num_cols, figsize=[80,30])\n",
    "    f.tight_layout()\n",
    "    a = [os.path.join(output_path, image) for image in os.listdir(output_path) \n",
    "         if os.path.splitext(image)[1].lower() in valid_image_ext]\n",
    "    for idx, img_path in enumerate(a[:num_images]):\n",
    "        col_id = idx % num_cols\n",
    "        row_id = idx / num_cols\n",
    "        img = plt.imread(img_path)\n",
    "        axarr[row_id, col_id].imshow(img) "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# Visualizing the first 12 images.\n",
    "OUTPUT_PATH = 'tlt_infer_testing/images_annotated' # relative path from $USER_EXPERIMENT_DIR.\n",
    "COLS = 4 # number of columns in the visualizer grid.\n",
    "IMAGES = 8 # number of images to visualize.\n",
    "\n",
    "visualize_images(OUTPUT_PATH, num_cols=COLS, num_images=IMAGES)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 9. Deploy! <a class=\"anchor\" id=\"head-9\"></a>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!mkdir -p $USER_EXPERIMENT_DIR/experiment_dir_final\n",
    "# Removing a pre-existing copy of the etlt if there has been any.\n",
    "import os\n",
    "output_file=os.path.join(os.environ['USER_EXPERIMENT_DIR'],\n",
    "                         \"experiment_dir_final/resnet18_detector.etlt\")\n",
    "if os.path.exists(output_file):\n",
    "    os.system(\"rm {}\".format(output_file))\n",
    "!tlt-export detectnet_v2 \\\n",
    "            -m $USER_EXPERIMENT_DIR/experiment_dir_retrain/weights/resnet18_detector_pruned.tlt \\\n",
    "            -o $USER_EXPERIMENT_DIR/experiment_dir_final/resnet18_detector.etlt \\\n",
    "            -k $KEY"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print('Exported model:')\n",
    "print('------------')\n",
    "!ls -lh $USER_EXPERIMENT_DIR/experiment_dir_final"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### A. Int8 Optimization <a class=\"anchor\" id=\"head-9-1\"></a>\n",
    "DetectNet_v2 model supports int8 inference mode in TRT. In order to use int8 mode, we must calibrate the model to run 8-bit inferences. This involves 2 steps\n",
    "\n",
    "* Generate calibration tensorfile from the training data using tlt-int8-tensorfile\n",
    "* Use tlt-export to generate int8 calibration table.\n",
    "\n",
    "*Note: For this example, we generate a calibration tensorfile containing 10 batches of training data.\n",
    "Ideally, it is best to use atleast 10-20% of the training data to calibrate the model. The more data provided during calibration, the closer int8 inferences are to fp32 inferences.*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!tlt-int8-tensorfile detectnet_v2 -e $SPECS_DIR/detectnet_v2_retrain_resnet18_kitti.txt \\\n",
    "                                  -m 40 \\\n",
    "                                  -o $USER_EXPERIMENT_DIR/experiment_dir_final/calibration.tensor"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!rm -rf $USER_EXPERIMENT_DIR/experiment_dir_final/resnet18_detector.etlt\n",
    "!rm -rf $USER_EXPERIMENT_DIR/experiment_dir_final/calibration.bin\n",
    "!tlt-export detectnet_v2 \\\n",
    "            -m $USER_EXPERIMENT_DIR/experiment_dir_retrain/weights/resnet18_detector_pruned.tlt \\\n",
    "            -o $USER_EXPERIMENT_DIR/experiment_dir_final/resnet18_detector.etlt \\\n",
    "            -k $KEY  \\\n",
    "            --cal_data_file $USER_EXPERIMENT_DIR/experiment_dir_final/calibration.tensor \\\n",
    "            --data_type int8 \\\n",
    "            --batches 20 \\\n",
    "            --batch_size 4 \\\n",
    "            --max_batch_size 4\\\n",
    "            --engine_file $USER_EXPERIMENT_DIR/experiment_dir_final/resnet18_detector.trt.int8 \\\n",
    "            --cal_cache_file $USER_EXPERIMENT_DIR/experiment_dir_final/calibration.bin \\\n",
    "            --verbose"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### B. Generate TensorRT engine <a class=\"anchor\" id=\"head-9-2\"></a>\n",
    "Verify engine generation using the `tlt-converter` utility included with the docker.\n",
    "\n",
    "The `tlt-converter` produces optimized tensorrt engines for the platform that it resides on. Therefore, to get maximum performance, please instantiate this docker and execute the `tlt-converter` command, with the exported `.etlt` file and calibration cache (for int8 mode) on your target device. The converter utility included in this docker only works for x86 devices, with discrete NVIDIA GPU's. \n",
    "\n",
    "For the jetson devices, please download the converter for jetson from the dev zone link [here](https://developer.nvidia.com/tlt-converter). \n",
    "\n",
    "If you choose to integrate your model into deepstream directly, you may do so by simply copying the exported `.etlt` file along with the calibration cache to the target device and updating the spec file that configures the `gst-nvinfer` element to point to this newly exported model. Usually this file is called `config_infer_primary.txt` for detection models and `config_infer_secondary_*.txt` for classification models."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!tlt-converter $USER_EXPERIMENT_DIR/experiment_dir_final/resnet18_detector.etlt \\\n",
    "               -k $KEY \\\n",
    "               -c $USER_EXPERIMENT_DIR/experiment_dir_final/calibration.bin \\\n",
    "               -o output_cov/Sigmoid,output_bbox/BiasAdd \\\n",
    "               -d 3,544,960 \\\n",
    "               -i nchw \\\n",
    "               -m 64 \\\n",
    "               -t int8 \\\n",
    "               -e $USER_EXPERIMENT_DIR/experiment_dir_final/resnet18_detector.trt \\\n",
    "               -b 4"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 10. Verify Deployed Model <a class=\"anchor\" id=\"head-10\"></a>\n",
    "Verify the exported model by visualizing inferences on TensorRT.\n",
    "In addition to running inference on a `.tlt` model in [step 8](#head-8), the `tlt-infer` tool is also capable of consuming the converted `TensorRT engine` from [step 9.B](#head-9-2).\n",
    "\n",
    "*If after int-8 calibration the accuracy of the int-8 inferences seem to degrade, it could be because the there wasn't enough data in the calibration tensorfile used to calibrate thee model or, the training data is not entirely representative of your test images, and the calibration maybe incorrect. Therefore, you may either regenerate the calibration tensorfile with more batches of the training data, and recalibrate the model, or calibrate the model on a few images from the test set. This may be done using `--cal_image_dir` flag in the `tlt-export` tool. For more information, please follow the instructions in the USER GUIDE."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### A. Inference using TensorRT engine <a class=\"anchor\" id=\"head-10-1\"></a>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!tlt-infer detectnet_v2 -e $SPECS_DIR/detectnet_v2_inference_kitti_etlt.txt \\\n",
    "                        -o $USER_EXPERIMENT_DIR/etlt_infer_testing \\\n",
    "                        -i $DATA_DOWNLOAD_DIR/test_images \\\n",
    "                        -k $KEY"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# visualize the first 12 inferenced images.\n",
    "OUTPUT_PATH = 'etlt_infer_testing/images_annotated' # relative path from $USER_EXPERIMENT_DIR.\n",
    "COLS = 4 # number of columns in the visualizer grid.\n",
    "IMAGES = 8 # number of images to visualize.\n",
    "\n",
    "visualize_images(OUTPUT_PATH, num_cols=COLS, num_images=IMAGES)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.11"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
